Aspects of the disclosure relate to the detection and utilization of object orientation associated with images captured by mobile devices. According to embodiments of the present disclosure, the detected orientation can be utilized in various ways. One example is the automatic control of the rotation of a display presented to a user by a mobile device. Assessing appropriate display rotation can be a challenging task. Current techniques are lacking in many ways in terms of both accuracy and computational efficiency. For instance, a simple auto display rotation feature commonly implemented in mobile devices utilizes the direction of gravity, as measured using an accelerometer, to control the rotation of the display presented to the user (e.g., “portrait” v. “landscape” rotation). However, proper display rotation sometimes cannot be accurately determined by simply using the direction of gravity. For example, if the mobile device is laid flat on a table with the display facing up toward the sky, the gravity vector is pointed in a direction perpendicular to the plane of the display and therefore provides no useful indication of the proper rotation of the display to present to the user. Oftentimes, the last orientation of the display before the device is laid down flat is kept in used, which may not be the appropriate rotation for proper viewing by the user. As another example, if the user holds a mobile device in front of his face while lying down on his side (e.g., on a bed), a gravity vector-controlled auto display rotation technique would generally result in incorrect display rotation from the perspective of the user. Here, the display may automatically rotate, incorrectly, with even small movements. This is because the system, using just the gravity vector, would only sense that the mobile device has moved from an upright position to a prone position and conclude that the display must be rotated (e.g., from “portrait” to “landscape” rotation) in response. However, the system would not sense that the user has also moved from an upright to a prone position, and therefore no rotation of the display would actually be needed. These and other shortcomings of existing systems highlight the need for improved techniques for image-related orientation detection and utilization.